<html>
<head><meta charset="utf-8"><title>CI is timing out · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html">CI is timing out</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="214259255"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214259255" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214259255">(Oct 22 2020 at 23:22)</a>:</h4>
<p>It looks like both try and regular bors builds are always timing out at the moment</p>



<a name="214259630"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214259630" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214259630">(Oct 22 2020 at 23:28)</a>:</h4>
<p>Can you close the tree?</p>



<a name="214259638"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214259638" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214259638">(Oct 22 2020 at 23:28)</a>:</h4>
<p>I probably don't have time to investigate tonight</p>



<a name="214260950"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214260950" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214260950">(Oct 22 2020 at 23:47)</a>:</h4>
<p>already closed it</p>



<a name="214261445"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214261445" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214261445">(Oct 22 2020 at 23:56)</a>:</h4>
<p>I imagine this is related to the recent bors migration, but I'm off for tonight -- <span class="user-mention" data-user-id="121055">@Pietro Albini</span> should hopefully be able to take a look tomorrow. I suspect we're restarting homu or losing the notification somewhere -- I'd need to do some deeper investigation, though, because the github hook seems to be just fine</p>



<a name="214288682"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214288682" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214288682">(Oct 23 2020 at 09:02)</a>:</h4>
<p>looking into this</p>



<a name="214288757"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214288757" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214288757">(Oct 23 2020 at 09:03)</a>:</h4>
<p><a href="/user_uploads/4715/pURYB05_vxum0cjoGmabF8dG/2020-10-23-11-03-03.png">2020-10-23-11-03-03.png</a> <br>
ok it seems like the azure dummy job is not reporting a success</p>
<div class="message_inline_image"><a href="/user_uploads/4715/pURYB05_vxum0cjoGmabF8dG/2020-10-23-11-03-03.png" title="2020-10-23-11-03-03.png"><img src="/user_uploads/4715/pURYB05_vxum0cjoGmabF8dG/2020-10-23-11-03-03.png"></a></div>



<a name="214289573"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214289573" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214289573">(Oct 23 2020 at 09:12)</a>:</h4>
<p>hmm, azure pipelines is not registering any check run</p>



<a name="214290139"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214290139" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214290139">(Oct 23 2020 at 09:18)</a>:</h4>
<p>ok, I have not figured out why, but azure pipelines just... stopped reporting status to github</p>



<a name="214290154"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214290154" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214290154">(Oct 23 2020 at 09:19)</a>:</h4>
<p>I'm stopping azure from gating our builds</p>



<a name="214290256"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214290256" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214290256">(Oct 23 2020 at 09:20)</a>:</h4>
<p>this unfortunately means stable releases will not work unless we revert the gate, but looking at <a class="stream" data-stream-id="241545" href="/#narrow/stream/241545-t-release">#t-release</a> there doesn't seem a need to do one</p>



<a name="214290990"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214290990" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214290990">(Oct 23 2020 at 09:28)</a>:</h4>
<p>reopened the tree</p>



<a name="214291194"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214291194" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214291194">(Oct 23 2020 at 09:30)</a>:</h4>
<p>whelp the queue looks bad</p>



<a name="214291277"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214291277" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Yuki Okushi <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214291277">(Oct 23 2020 at 09:31)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/CI.20is.20timing.20out/near/214291194">said</a>:</p>
<blockquote>
<p>whelp the queue looks bad</p>
</blockquote>
<p>created big rollup...</p>



<a name="214301504"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214301504" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214301504">(Oct 23 2020 at 11:33)</a>:</h4>
<p>/me wonders how many perf regressions will hide themselves</p>



<a name="214302148"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214302148" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Yuki Okushi <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214302148">(Oct 23 2020 at 11:41)</a>:</h4>
<p>tried to pick up PRs that don't seem to affect perf but yeah sorry about that if it occurs :(</p>



<a name="214304496"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214304496" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214304496">(Oct 23 2020 at 12:03)</a>:</h4>
<p>Got a 503 for <a href="http://bors.rust-lang.org">bors.rust-lang.org</a>.<br>
Edit: works now.</p>



<a name="214306563"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214306563" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mara <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214306563">(Oct 23 2020 at 12:27)</a>:</h4>
<p>Looks like it forgot which PRs are mergable.</p>



<a name="214307693"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214307693" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214307693">(Oct 23 2020 at 12:38)</a>:</h4>
<p>uuh it seems to have desynced <a href="https://github.com/rust-lang/rust/pull/78279">https://github.com/rust-lang/rust/pull/78279</a></p>



<a name="214308126"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214308126" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214308126">(Oct 23 2020 at 12:43)</a>:</h4>
<p>triggered a resync, looks like that takes a few minutes</p>



<a name="214310276"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214310276" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214310276">(Oct 23 2020 at 13:04)</a>:</h4>
<p>looks like <a href="https://github.com/rust-lang/rust/pull/77015">https://github.com/rust-lang/rust/pull/77015</a> is now stuck, I'll try creating a rollup</p>



<a name="214310853"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214310853" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214310853">(Oct 23 2020 at 13:09)</a>:</h4>
<p>that didn't seem to work</p>



<a name="214335244"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335244" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335244">(Oct 23 2020 at 16:31)</a>:</h4>
<p><span class="user-group-mention" data-user-group-id="2943">@T-infra</span> FYI: The bors queue is still stuck <a href="https://bors.rust-lang.org/queue/rust">https://bors.rust-lang.org/queue/rust</a></p>



<a name="214335279"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335279" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335279">(Oct 23 2020 at 16:32)</a>:</h4>
<p>groan</p>



<a name="214335308"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335308" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335308">(Oct 23 2020 at 16:32)</a>:</h4>
<p><span class="user-mention" data-user-id="121055">@Pietro Albini</span> I think we need to revert the bors migration, it's causing too much trouble tbh</p>



<a name="214335313"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335313" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335313">(Oct 23 2020 at 16:32)</a>:</h4>
<p>will look in 5 minutes</p>



<a name="214335328"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335328" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335328">(Oct 23 2020 at 16:32)</a>:</h4>
<p>unless we have fixes in mind this is just not tenable</p>



<a name="214335353"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335353" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335353">(Oct 23 2020 at 16:32)</a>:</h4>
<p>agreed :(</p>



<a name="214335382"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335382" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335382">(Oct 23 2020 at 16:32)</a>:</h4>
<p>I'll try to look at logs now</p>



<a name="214335414"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335414" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335414">(Oct 23 2020 at 16:33)</a>:</h4>
<blockquote>
<p>fatal: Unable to create '/efs/git-cache/rust-lang/rust/.git/index.lock': File exists.</p>
</blockquote>



<a name="214335472"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335472" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335472">(Oct 23 2020 at 16:33)</a>:</h4>
<p>hm, okay, so we definitely should not be putting the git cache in efs I think, since that's just going to cause conflicts when we try to restart homu</p>



<a name="214335503"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335503" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335503">(Oct 23 2020 at 16:33)</a>:</h4>
<p>I suspect probably the right thing is to store it locally, without submodules it should definitely fit</p>



<a name="214335587"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335587" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335587">(Oct 23 2020 at 16:34)</a>:</h4>
<p><span aria-label="thumbs up" class="emoji emoji-1f44d" role="img" title="thumbs up">:thumbs_up:</span> remove the cache line from cfg.production.toml</p>



<a name="214335600"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335600" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335600">(Oct 23 2020 at 16:34)</a>:</h4>
<p>doing that</p>



<a name="214335929"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335929" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335929">(Oct 23 2020 at 16:37)</a>:</h4>
<p>I can babysit it for the rest of today at least, which is roughly 6 hours I think</p>



<a name="214335940"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335940" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335940">(Oct 23 2020 at 16:37)</a>:</h4>
<p><a href="https://github.com/rust-lang/homu/pull/111">https://github.com/rust-lang/homu/pull/111</a></p>



<a name="214335997"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214335997" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214335997">(Oct 23 2020 at 16:37)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> ok merged it</p>



<a name="214336036"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214336036" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214336036">(Oct 23 2020 at 16:38)</a>:</h4>
<p>the reason why I implemented the cache was because it took a couple of minutes to clone the rustc repository</p>



<a name="214336114"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214336114" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214336114">(Oct 23 2020 at 16:38)</a>:</h4>
<p>...after each restart, which caused bors to seem idle while it was cloning the repo before doing the merge</p>



<a name="214336132"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214336132" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214336132">(Oct 23 2020 at 16:38)</a>:</h4>
<p>hm</p>



<a name="214336149"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214336149" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214336149">(Oct 23 2020 at 16:39)</a>:</h4>
<p>honestly I'd rather it idles for a bit</p>



<a name="214336151"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214336151" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214336151">(Oct 23 2020 at 16:39)</a>:</h4>
<p>but that's acceptable compared to having bors deadlock</p>



<a name="214336175"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214336175" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214336175">(Oct 23 2020 at 16:39)</a>:</h4>
<p>I think I can implement a "bors is cloning the repo" banner</p>



<a name="214336181"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214336181" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214336181">(Oct 23 2020 at 16:39)</a>:</h4>
<p>it shouldn't be that difficult</p>



<a name="214336208"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214336208" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214336208">(Oct 23 2020 at 16:39)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/242791-t-infra/topic/CI.20is.20timing.20out/near/214336149">said</a>:</p>
<blockquote>
<p>honestly I'd rather it idles for a bit</p>
</blockquote>
<p>if this is the tradeoff absolutely</p>



<a name="214338318"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338318" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338318">(Oct 23 2020 at 16:56)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> cloning rust-lang/rust took 12 minutes</p>



<a name="214338334"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338334" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338334">(Oct 23 2020 at 16:56)</a>:</h4>
<p>I don't understand</p>



<a name="214338351"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338351" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338351">(Oct 23 2020 at 16:56)</a>:</h4>
<p>Are we just super bandwidth limited?</p>



<a name="214338375"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338375" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338375">(Oct 23 2020 at 16:57)</a>:</h4>
<p>I think it's because I assumed 1/4 of a CPU core would be enough for core</p>



<a name="214338403"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338403" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338403">(Oct 23 2020 at 16:57)</a>:</h4>
<p>but tbh git likes paralellism while it clones a repo</p>



<a name="214338474"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338474" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338474">(Oct 23 2020 at 16:58)</a>:</h4>
<p><a href="/user_uploads/4715/I6DJpWzwsusTXf8xwH6e7EjS/2020-10-23-18-57-57.png">2020-10-23-18-57-57.png</a> <br>
yep cpu graph</p>
<div class="message_inline_image"><a href="/user_uploads/4715/I6DJpWzwsusTXf8xwH6e7EjS/2020-10-23-18-57-57.png" title="2020-10-23-18-57-57.png"><img src="/user_uploads/4715/I6DJpWzwsusTXf8xwH6e7EjS/2020-10-23-18-57-57.png"></a></div>



<a name="214338497"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338497" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338497">(Oct 23 2020 at 16:58)</a>:</h4>
<p>okay so sounds like we should bump to at least a single full cpu</p>



<a name="214338536"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338536" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338536">(Oct 23 2020 at 16:58)</a>:</h4>
<p>doing that</p>



<a name="214338539"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338539" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338539">(Oct 23 2020 at 16:58)</a>:</h4>
<p>I think it's probably misleading (not sure) but a -j1 git clone locally takes roughly 1 minute</p>



<a name="214338566"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338566" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338566">(Oct 23 2020 at 16:59)</a>:</h4>
<blockquote>
<p>The number of submodules fetched at the same time</p>
</blockquote>
<p>ah okay so that's highly misleading</p>



<a name="214338702"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338702" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338702">(Oct 23 2020 at 17:00)</a>:</h4>
<p>TIL bors is so resource hungry</p>



<a name="214338730"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338730" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338730">(Oct 23 2020 at 17:00)</a>:</h4>
<p>I guess we never noticed since RCS is not a tiny instance</p>



<a name="214338839"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338839" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338839">(Oct 23 2020 at 17:01)</a>:</h4>
<p>we might also need more ram, it looks like RCS idled at 0.66 GB</p>



<a name="214338867"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338867" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338867">(Oct 23 2020 at 17:02)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> I already bumped that to 2GB a while ago</p>



<a name="214338954"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214338954" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214338954">(Oct 23 2020 at 17:02)</a>:</h4>
<p>At this rate bors is going to be more expensive than a crater instance</p>



<a name="214339026"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339026" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339026">(Oct 23 2020 at 17:03)</a>:</h4>
<p>that's kinda sad</p>



<a name="214339080"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339080" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339080">(Oct 23 2020 at 17:03)</a>:</h4>
<p>maybe something we should mention to the bors-rs folks</p>



<a name="214339094"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339094" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339094">(Oct 23 2020 at 17:03)</a>:</h4>
<p>so, to summarize bors on ecs's problems:</p>
<ul>
<li>tonight's outage: unrelated to ECS, we simply lost the Azure Pipelines webhooks</li>
<li>bors deadlocking: we're fixing that right now</li>
<li>503s: no clue yet</li>
</ul>



<a name="214339101"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339101" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339101">(Oct 23 2020 at 17:03)</a>:</h4>
<p>am I missing something?</p>



<a name="214339184"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339184" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339184">(Oct 23 2020 at 17:04)</a>:</h4>
<p>my bet on 503s is restarts personally, but we need to correlate restarts and those</p>



<a name="214339196"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339196" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339196">(Oct 23 2020 at 17:04)</a>:</h4>
<p>I wish we had heroku like dashboard</p>



<a name="214339214"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339214" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339214">(Oct 23 2020 at 17:04)</a>:</h4>
<p>it's definitely restarts</p>



<a name="214339258"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339258" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339258">(Oct 23 2020 at 17:04)</a>:</h4>
<p>we might be able to bump up the "wait" time for the health endpoint</p>



<a name="214339275"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339275" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339275">(Oct 23 2020 at 17:04)</a>:</h4>
<p>like, I had a 503 while it was restarting after a deploy</p>



<a name="214339302"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339302" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339302">(Oct 23 2020 at 17:05)</a>:</h4>
<p>after bumping the CPU I'll sift through the logs</p>



<a name="214339336"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339336" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339336">(Oct 23 2020 at 17:05)</a>:</h4>
<p><a href="http://Docs.rs">Docs.rs</a> has the same problem with restarts FWIW</p>



<a name="214339369"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339369" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339369">(Oct 23 2020 at 17:06)</a>:</h4>
<p>Fortunately our restarts take &lt; 10 seconds so it's not a giant deal</p>



<a name="214339421"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339421" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339421">(Oct 23 2020 at 17:06)</a>:</h4>
<p><span class="user-mention" data-user-id="232545">@Joshua Nelson</span> yep, but like, ecs should've solved this for us</p>



<a name="214339446"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339446" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339446">(Oct 23 2020 at 17:06)</a>:</h4>
<p>since it's supposed to start another instance in parallel, wait for it to become healthy, and then redirect traffic to it</p>



<a name="214339508"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339508" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339508">(Oct 23 2020 at 17:07)</a>:</h4>
<p>hm so I'm seeing <a href="/user_uploads/4715/Pr9R49B896ePdcLbUyePSNey/image.png">image.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/Pr9R49B896ePdcLbUyePSNey/image.png" title="image.png"><img src="/user_uploads/4715/Pr9R49B896ePdcLbUyePSNey/image.png"></a></div>



<a name="214339533"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339533" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339533">(Oct 23 2020 at 17:07)</a>:</h4>
<p>which looks like the <em>running</em> bors went down</p>



<a name="214339562"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339562" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339562">(Oct 23 2020 at 17:07)</a>:</h4>
<p>that'd explain it</p>



<a name="214339670"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214339670" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214339670">(Oct 23 2020 at 17:08)</a>:</h4>
<p>going to try and correlate that with logs</p>



<a name="214340109"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340109" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340109">(Oct 23 2020 at 17:12)</a>:</h4>
<p>e199e30b-fa4f-4a54-81ea-9422af2b1ebe 2020-10-23 12:56:44 -0400 service bors (port 80) is unhealthy in target-group bors due to (reason Request timed out).</p>



<a name="214340164"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340164" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340164">(Oct 23 2020 at 17:13)</a>:</h4>
<p>my guess is it's going to be super annoying to find that</p>



<a name="214340301"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340301" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340301">(Oct 23 2020 at 17:14)</a>:</h4>
<p>it looks like that means we had 2 successive request time outs, which means 2 requests taking &gt;5 seconds</p>



<a name="214340365"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340365" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340365">(Oct 23 2020 at 17:15)</a>:</h4>
<p>should we just... increase the timeout?</p>



<a name="214340379"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340379" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340379">(Oct 23 2020 at 17:15)</a>:</h4>
<p>bors is single-thread tho, so that's not <em>too</em> surprising</p>



<a name="214340391"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340391" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340391">(Oct 23 2020 at 17:15)</a>:</h4>
<p>5 <em>seconds</em> though?!</p>



<a name="214340417"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340417" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340417">(Oct 23 2020 at 17:15)</a>:</h4>
<p>like what is it even doing</p>



<a name="214340419"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340419" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340419">(Oct 23 2020 at 17:15)</a>:</h4>
<p>I think it depends on the request</p>



<a name="214340478"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340478" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340478">(Oct 23 2020 at 17:16)</a>:</h4>
<p>this is specifically just /health requests</p>



<a name="214340499"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340499" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340499">(Oct 23 2020 at 17:16)</a>:</h4>
<p>yes, but bors is single threaded</p>



<a name="214340516"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340516" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340516">(Oct 23 2020 at 17:16)</a>:</h4>
<p>if there is a heavy webhook being processed everything is stuck</p>



<a name="214340541"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340541" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340541">(Oct 23 2020 at 17:16)</a>:</h4>
<p>ah :/</p>



<a name="214340570"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340570" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340570">(Oct 23 2020 at 17:16)</a>:</h4>
<p>yeah I guess we can bump timeout to like 60 seconds or something</p>



<a name="214340575"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340575" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340575">(Oct 23 2020 at 17:17)</a>:</h4>
<p>and I don't want to even think about adding multiple threads to bors</p>



<a name="214340594"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340594" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340594">(Oct 23 2020 at 17:17)</a>:</h4>
<p>with all the global state it has</p>



<a name="214340609"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340609" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340609">(Oct 23 2020 at 17:17)</a>:</h4>
<p>and all the tests it <em>doesn't</em> have</p>



<a name="214340616"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340616" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340616">(Oct 23 2020 at 17:17)</a>:</h4>
<p>note also that github webhooks just vanish after 30 seconds timeouts</p>



<a name="214340731"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340731" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340731">(Oct 23 2020 at 17:18)</a>:</h4>
<p>bors is just so broken</p>



<a name="214340755"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214340755" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214340755">(Oct 23 2020 at 17:18)</a>:</h4>
<p>... so how is bors-rs coming?</p>



<a name="214341407"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341407" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341407">(Oct 23 2020 at 17:24)</a>:</h4>
<p><span class="user-mention" data-user-id="121055">@Pietro Albini</span> do you want to bump the timeout in terraform I guess</p>



<a name="214341423"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341423" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341423">(Oct 23 2020 at 17:24)</a>:</h4>
<p>or I can do it</p>



<a name="214341431"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341431" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341431">(Oct 23 2020 at 17:24)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> sure, I can do that</p>



<a name="214341441"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341441" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341441">(Oct 23 2020 at 17:24)</a>:</h4>
<p>ok thanks</p>



<a name="214341447"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341447" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341447">(Oct 23 2020 at 17:24)</a>:</h4>
<p>what do we do, 30 seconds? a minute?</p>



<a name="214341470"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341470" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341470">(Oct 23 2020 at 17:24)</a>:</h4>
<p>let's go for a minute</p>



<a name="214341505"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341505" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341505">(Oct 23 2020 at 17:25)</a>:</h4>
<p>and maybe if we can like 10 successive requests need to be bad for  unhealthy</p>



<a name="214341593"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341593" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341593">(Oct 23 2020 at 17:25)</a>:</h4>
<p>that'd be 10 minutes until it recycles the instance</p>



<a name="214341666"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341666" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341666">(Oct 23 2020 at 17:26)</a>:</h4>
<p>what do you mean by recycle?</p>



<a name="214341679"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341679" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341679">(Oct 23 2020 at 17:26)</a>:</h4>
<p>I mean we don't expect bors to die on us right?</p>



<a name="214341704"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341704" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341704">(Oct 23 2020 at 17:26)</a>:</h4>
<p>didn't the old setup on rcs just die permanently?</p>



<a name="214341732"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341732" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341732">(Oct 23 2020 at 17:26)</a>:</h4>
<p>I guess</p>



<a name="214341777"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341777" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341777">(Oct 23 2020 at 17:27)</a>:</h4>
<p>I'd kinda prefer to keep it on 2 health checks failing and if we get more reports of 503s we increase again</p>



<a name="214341784"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341784" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341784">(Oct 23 2020 at 17:27)</a>:</h4>
<p>ok</p>



<a name="214341795"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341795" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341795">(Oct 23 2020 at 17:27)</a>:</h4>
<p>it'd be great to get an alert on unhealthy hosts &gt; 1</p>



<a name="214341819"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341819" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341819">(Oct 23 2020 at 17:27)</a>:</h4>
<p>we need to hook up monitoring with aws cloudwatch</p>



<a name="214341913"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/CI%20is%20timing%20out/near/214341913" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/CI.20is.20timing.20out.html#214341913">(Oct 23 2020 at 17:28)</a>:</h4>
<p>ETOOMUCHSTUFFTODO</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>